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Executive Summary 
Evaluation of 4 Schools for Engineering 



With funding from the National Science Foundation, engineering schools at Northeastern 
University, Tufts University, Worcester Polytechnic Institute, and Boston University joined 
forces in an effort to increase the number of girls who develop an interest in science, 
technology, engineering, and mathematics (STEM) fields during the middle school years, as 
well as in STEM careers. The 4 schools for Women in Engineering (WIE) strategy was to 
form all-female teams to train teachers and to serve as in-class resources for them. STEM 
Teams developed gender equitable engineering units to be used in 8 th grade science 
classrooms and helped middle school teachers implement the new units for the engineering 
strand of the Massachusetts Frameworks. The strategy was implemented in eight public 
school districts in the greater Boston area. 

Evaluation Components 

The evaluation focused on how teachers and STEM team members experienced the program 
and what the impact of the intervention was on students. Teachers’ and STEM Team 
members’ perceptions were assessed through open-ended anonymous questionnaires 
completed after the implementation of the intervention. The impact of the intervention on 
students was assessed through comparing attitudes toward engineering, mathematics, science, 
and STEM fields and careers before and after the intervention. 



1. Teacher Outcomes 

In a paper and pencil survey with open-ended questions teachers were asked to reflect on the 
benefits of the project and the effectiveness of the STEM Teams, and for their 
recommendations for additional training, support, STEM Team composition and curricular 
changes. Integrating gender equity into their classroom teaching and activities was singled 
out as a highly effective component of the intervention. For the teachers participation in the 
STEM Teams involved exposure to expertise from actual engineers, graduate and 
undergraduate students, and industry representatives coupled with the training. Teachers 
reported that the training was an integral part of their participation and that it made a 
difference in both their understanding of engineering and in how to teach this subject 
effectively. For their students, the benefits teachers highlighted were exposure to role 
models, glimpses into the real world of work for engineers, and a curriculum that made it 
possible for all students to achieve up to a benchmark of proficiency rather than a 
competitive classroom environment where only a few students are engaged in the projects. 

2. Impact on STEM Team Members 

Each STEM Team was composed of a coordinator, teachers, faculty members, undergraduate 
and graduate students, and industry representatives. In group discussions and in response to 
an open-ended survey STEM Team members reported having enjoyed the regular meetings 
that kept the teams focused and remarked on the synergy generated by the diversity in team 
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membership. The ability to expose middle school students to a wide variety of practicing 
engineers was also mentioned as a strength of the program. Several respondents recognized 
the value of having undergraduate students in the classroom, which not only made some 
projects possible but also was instrumental in establishing good rapport with the 8 th grade 
students. Some team members also noted the challenges of recruiting and training 
undergraduates for the classroom. 

All in all, the evaluation showed that it is feasible to bring together individuals from different 
domains to work together toward enhancing the teaching of STEM fields, with a specific 
emphasis on engineering. 

3. Impact on Students 

The student evaluation was carried out by comparing answers students gave to the same 
questionnaire administered in the beginning of the school year (pre-test) and then at the end 
of the school year (post-test). The outcome of the intervention is reported both in terms of 
general program impact averaged over all students and also in terms of its effect on 
individual students. 



Program Level Outcomes 

On the whole, students indicated that they found the classroom activities “somewhat” or 
“very interesting,” and this was true both for students who were not particularly interested in 
pursuing careers in math, science, and engineering as well as for those who did. Attitudes 
toward science were more positive after the intervention, attitudes toward engineering were 
less positive, while attitudes toward math did not change. The fact that the science attitudes 
showed a significant increase over the course of the semester speaks to the overall efficacy of 
the curriculum, which was presented in science classrooms. 

Several explanations can address why the engineering attitudes were less positive following 
the intervention than before it. First, it may be that the use of attitudinal measures does not 
adequately reflect the actual learning that has taken place in math, science, and engineering 
and that testing actual knowledge might have resulted in more favorable findings. Another 
explanation is that for some students, a less favorable attitude toward engineering after the 
intervention may reflect a more realistic appraisal of their thinking about engineering after 
they became better acquainted with it through the intervention. Indeed, the individual level 
outcomes bear this out in that some students’ attitudes toward engineering, especially girls’, 
were more highly influenced by the intervention than their attitudes toward science or math. 

Overall, when changes in girls’ attitudes toward STEM fields and careers were compared to 
those in boys, more girls reported a positive attitude change toward careers in engineering 
and chemistry than did boys. For girls, which school they attended (and thereby which 
teacher taught them, because only one teacher participated in most schools) was related to 
whether the intervention was associated with greater attitude changes toward both STEM 
fields and STEM careers, whereas it made no difference for boys. This finding is based on a 
small number of schools and needs to be viewed as tentative. 




4 



Individual Level Student Outcomes 

In spite of the fact that students’ overall scores on attitudes toward engineering did not 
become more positive after the intervention, there were a large number of individual students 
whose scores increased — that is, their attitudes toward STEM fields or careers became more 
positive — between pre- and post-test. We examined the impact of the intervention on 
individual students using regression analyses which yield information on what variables best 
predict whether a student’s scores will increase or decrease. The results show that students 
who started out with lower positive attitudes toward both STEM fields and STEM careers 
made more gains after the intervention than students who started out with more positive 
attitudes. Moreover, the intervention had a greater impact on female students’ attitudes 
toward engineering as a field of study as well as on choosing engineering as a career than it 
did on male students’ attitudes. 

We interpret the finding that the intervention had more influence on girls’ attitudes (both 
positive and negative) toward engineering as a field and a career in terms of the possibility 
that girls were relatively less well informed about engineering hence had greater openness to 
new information on this topic. Indeed, girls’ attitudes were influenced at the same level as 
boys’ on biology and chemistry as fields of study and careers, which are better known fields. 
We believe that the intervention was able to influence girls more because, due to a lack of 
previous experience or knowledge, girls had fewer preconceived notions about engineering. 
This interpretation is bolstered by the finding that before the intervention fewer girls reported 
knowing engineers than did boys. 



Conclusions 

The evaluation shows that the 4 Schools for WIE project had the intended outcomes: 

• Capacity building among 8 th grade science teachers for increasing their confidence to 
teach engineering; 

• Demonstration of the feasibility of forming well-functioning STEM teams that include 
middle school teachers, academics, graduate and undergraduate students, and industry 
representatives; 

• Program level increase in positive attitudes toward science as a field of study (but not in 
engineering or mathematics); 

• Individual level impact of the STEM Team strategy on girls’ attitudes toward engineering 
both as a field of study and as a career to a greater extent than on boys’ attitudes. 

An unexpected but welcome finding was that the intervention was associated with greater 
improvement in the attitudes of students who started out with relatively low scores on 
attitudes toward STEM fields and careers than those who started out with more positive 
attitudes. The greater benefit of the intervention on students who showed the most positive 
change relative to where they started out in the beginning of the academic year and who had 
relatively less positive attitudes toward STEM fields and careers was a finding that was true 
for both girls and boys. This suggests that the intervention has the potential to benefit those 
who hold less positive views of STEM rather than merely making students who already hold 
positive attitudes even more positive. 
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All in all, the STEM team intervention succeeded in its goal to improve teachers’ capacity to 
teach the new engineering strand in the 8 th grade science curriculum. It succeeded partially in 
terms of improving students’ attitudes toward STEM fields: all students’ attitudes toward 
science as a field of study improved, but not toward engineering or mathematics. Perhaps 
most significantly, it resulted in more girls than boys coming to view engineering and 
chemistry as possible careers for themselves and had a larger impact on girls than it did on 
boys. Moreover, the context of school or teacher was related to changes in girls’ attitudes 
toward engineering and mathematics as a field of study and engineering as a career but not 
that of boys. 
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4 Schools for WIE (Women in Engineering) 
Evaluation Report 



Background 

With funding from the National Science Foundation, engineering schools at Northeastern 
University, Tufts University, Worcester Polytechnic Institute, and Boston University joined 
forces in an effort to increase the number of girls who develop an interest in science, 
technology, engineering, and mathematics (STEM) areas during the middle school years as 
well as in STEM careers. As an important step toward achieving this mission, the four 
schools designed a unique intervention system composed of trained STEM teams consisting 
of mostly female engineering faculty, students, teachers, and industry representatives as 
practitioners (with only a few males). Each engineering school developed its own STEM 
team to train middle school teachers about engineering and gender-inclusive practices and to 
serve as role models in the middle school classrooms. In addition to paying attention to 
gender equity, the STEM teams designed lesson plans that met the requirements of Science 
and Technology/Engineering Curriculum Frameworks in the Commonwealth of 
Massachusetts. 1 The 8th grade science classrooms in eight public schools in the greater 
Boston area used the engineering lesson units and benefited from the teams’ help in 
implementation of the engineering strand of the Massachusetts Frameworks. For more 
information on the design of the project go to www.steamteams.org. 

Evaluation Design 

The evaluation focused on outcomes for teachers, STEM Team members and, more 
importantly, on the impact of the intervention on students. Teacher and STEM Team 
outcomes were assessed through an open-ended anonymous questionnaire. The impact of the 
intervention on students was assessed through comparing attitudes toward engineering, 
mathematics, science, and STEM careers before and after the intervention. All questionnaires 
were extensively field-tested in the first and second year of the program’s operation. 



Teacher Evaluation 



Teacher Questionnaire 

Teacher outcomes were assessed at the end of the 2003-2004 school year and again at the 
final feedback session held on September 30, 2005. The first survey asked teachers to reflect 
on the goals of the overall project and the effectiveness of the STEM Teams and to suggest 
recommendations for additional training, support, STEM Team composition and curricular 
changes (a copy of the Teacher Questionnaire and Report is in Appendix A). 

The third-year survey asked teachers about what aspects of the STEM Team approach 
worked and didn’t work for them. 



1 Beginning in the fall of 2001, Massachusetts was the first state in the nation to introduce 
engineering as part of its mandated PreK-12 education frameworks. 
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Teacher Outcomes 

Compared to the more extensive survey teachers completed at the end of the school year 
2003-2004, only two questions were asked of the teachers implementing the curriculum at 
the conclusion of project. The teachers were asked to reflect about: 1) which aspects of the 
STEM Team approach worked for them; and 2) which aspects did not work for them. All six 
teachers responding to the survey had been involved in the project from its inception. 

One of the ways in which the STEM Team approach appears to work effectively is by 
increasing access to expertise from faculty and undergraduate and graduate students plus the 
extra classroom support for 8 th grade students and the teacher. Also mentioned was the 
support received through the provision of resources including equipment, supplies, access to 
information. One teacher also commented positively on the intensity and frequency of the 
university involvement. 

For some teachers a key element in the success of the undertaking was the expertise 
contributed by each STEM Team member. For example, the engineers provided their 
expertise in science, the graduate and undergraduate students brought to the table the most 
current research findings, the industry representative provided real-life, fresh perspectives 
from their field experience, and middle school teachers provided knowledge of how to 
approach and teach young adolescents. 



Some teachers commented on the positive effects of providing role models for students. 

Students benefited from positive role models... women as engineers in the classroom. 
Many young women in my class expressed a strong interest in pursuing careers in 
biotechnology. 

The sharing of current research by the undergrads gave my students a real sense of 
how biotechnology helps people. . . they wanted to pursue a career where they could 
help make someone else’s life better. 

My students look forward to our projects and having engineers in the classroom. 
Many of my students do not have college role models, nor engineers in their lives. 
The exposure to engineering and what they do and who they are was wonderful. 

It also provided a mentor/mentee level to both myself and my students. 

Several teachers commented on how enjoyable their involvement in the project was for them 
and how much it contributed to their own learning. 

I have been part of many partnerships with higher education institutions and this has 
been one of the most enjoyable and resulted in deep learning. 
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I grew in my own confidence in terms of understanding molecular biology and 
biotechnology. 

Project based approach was effective and fun! 

This was the first time in my career that I have actually worked with “real” people in 
a career field. They increased my knowledge and ability to teach engineering. 

The collaboration among our team was probably the most influential development in 
all my education training. I had never been exposed to engineering and the old saying 
that “the teacher learns the most” was very true. 

I learned to think as an engineer and now it is simply a part of my planning. 

One of the goals of the project was to address gender issues in math, science, technology and 
engineering in mixed-gender classrooms. According to the teacher comments, both the 
curriculum and the training succeeded in achieving positive outcomes in gender equity. 

This is of particular concern in the classroom where both boys and girls are present. 
Boys tend to grab the materials first, build first and test first, before their female 
counterparts have had sufficient time to consider the possibilities and make a plan of 
action. But long term design challenges required students to apply their understanding 
of science in a very explicit way. This is often not the case in engineering design 
challenges. 

It helped me as a teacher think about gender in the classroom and provided me with 
tools to help both genders succeed. . . projects where everyone has to reach a certain 
benchmark as opposed to competitive winning situations. 



Additional comments indicated that students became knowledgeable and developed a deeper 
understanding of the design process as evidenced in their ability to apply it across contexts. 
The program also succeeded in avoiding many of the pitfalls of other currently used 
engineering curricula. 

Students saw themselves as scientists. They were in fact engineers and scientists. 
Their belief in themselves as smart and capable grew tremendously. 

The curriculum avoided many of the weaknesses in currently marketed engineering 
curricula, which while some experiments work, they do not result in a greater 
understanding or application of scientific content or principles. 

Compared with the extensive accolades the six teachers gave the program, when asked what 
about the STEM Team approach did not work for them, they provided only very brief 
comments. One major concern voiced by two teachers was the issue of “timing.” 
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Timing was tricky at times. . . in the classroom. . . to come to STEM team meetings. 
Lack of time is a huge constraint in all areas of life, especially in the classroom. 

The only negative that I found in the program was the scheduling. It is very difficult 
to align 8 people. Every professional was as flexible as they could be: for the students 
[graduate and undergraduate students] it was a major problem. 

Another teacher voiced concern about the messages that were sent when there was a lack of 
growth in student attitudes. She felt that activities during the fall workshops were geared 
towards “fixing” teachers rather than a reevaluation of whether the structure of the program 
might not also be an effective strategy. 



STEM Team Members’ Evaluation 

In addition to teachers, the third year survey also asked other STEM Team members about 
the efficacy of the STEM Team approach. Respondents ranged from STEM Team 
coordinators, college faculty members, graduate and undergraduate students to industry 
representatives. 

One of the most positively perceived aspects of the approach was the diversity of the teams 
and the regular meetings that served to keep the teams focused. The ability to expose middle 
school students to a wide variety of practicing engineers was also mentioned as strength of 
the program. Several respondents recognized the value of having undergraduate students in 
the classroom, which not only made some projects possible but was also key in establishing 
good rapport with the 8 th grade students. The challenges of recruiting and training 
undergraduates for the classroom were also noted. 

Faculty members spoke to the value of full team training as did the industry representatives, 
who commented on the importance of the training in grounding everyone on the team, 
providing an opportunity to meet all the players and form relationships that would last 
beyond the program. One of the industry representatives commented on the freedom the 
teams had in developing their own projects, which increased the level of the team’s 
commitment to the undertaking. 

I thought the most valuable part of the program was bringing in real engineers to 

speak to the students We had an environmental engineer from Deer Island, a 

professor who works on solar cells, an engineer who works at a local biomedical 
(device) company, and a grad student who develops programs coming to speak to the 
students.... I think having a diverse group of speakers is really important. Maybe 
students will see a person they can identify with or a project that sound really cool 
and they will think, “I can do that.” Especially if the student doesn’t have a parent 
who is an engineer. . . they might never consider [engineering] as a career. 
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Partnering university, middle schools and industry so that each could manage “part” 
of the program they are most capable of; it would have been very difficult for any one 
partner to coordinate and pull off [the] entire program. 

Allowing each STEM team to create and implement their own projects gave 
ownership and a higher level of commitment from the STEM teams. 

I learned a lot about thinking about gender in the classroom; inspiring discussions, 
ideas. . . . Working with the teachers was very good. . .so that the projects were at the 
appropriate level for the students. 

Several comments related to the effect of the projects on the middle school students. One 
respondent commented on a specific project that provided the students, particularly the girls, 
with an opportunity to use real equipment and feel like real scientists. Another STEM Team 
member felt that the students were exposed to and learned about engineering principles that 
interested them, without necessarily realizing they were learning them. 

I think some of them [the girls] will now consider a career in engineering The 

girls excelled at the project because it required taking precise measurements and 
carefully adding liquids, etc. 

I believe the students were enriched and inspired by the program. 

In terms of what did not work in the STEM team approach, similar to the teachers, other 
team members had relatively few comments. One of the few frustrations mentioned was 
finding undergraduates who had the time to participate in the classrooms. A suggestion was 
made to pair an engineering student and an education student. These students would work in 
pairs and learn from each other how to teach science. 

We asked a lot of the [undergraduate and graduate] students and they were given no 
real training on how to act/interact with middle school students. 

The other issue was one of time, a pressure felt by almost all team members. One of the 
industry representatives noted the need for extensive coordination in order to make the 
projects work and that while the lesson plans were delivered, the activities were not 
consistent across middle schools. 

Everyone felt very pressed for time. We (the non-teachers) wanted to be in the 
classrooms more, but we each had too many other responsibilities. 

Letting students know in advance the amount of time expected of them was another 
recommendation. 

One respondent raised concerns about the use of prizes and the idea of winning rather than 
learning. 
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The part that did not work for me was the association with prizes or awards for each 
project we did. I think the students associated winning and prizes with being 
successful rather than learning and the knowledge and skills that they gained. I think 
if the awards were taken away, and [not] choosing students as the best or not the best, 
the students would appreciate the projects that they did more, rather than focusing on 
the prizes or the place they came in compared to other students. 

A key component of the STEM Team’s success was due to the work of the coordinator of 
each team who took responsibility for the organizational, administrative, and also curricular 
aspects of the strategy and worked closely with the teachers. 

We had fabulous teachers and that is what would make or break this approach. We 
also had a great coordinator who kept everything organized. 



Student Evaluation 



Evaluation Design 

The evaluation was carried out by comparing answers students gave to the same 
questionnaire administered in the beginning of the school year (pre-test) and then at the end 
of the school year (post-test).' The pre-test survey was administered between 9/22/04 and 
11/18/04; the post-test survey between 4/11/05 and 4/28/05. The retention rate between 
survey administrations was 88%.' Questionnaires were administered by the STEM Team 
coordinators during the science class period. Most students were able to complete the 
questionnaires within 20 minutes. 

The impact of the intervention is reported in terms of a general program impact averaged 
over all students and also in terms of its effect on individual students. The former is 
important for understanding program level effects. The latter provides information on the 
ways in which individual students were influenced by the intervention. This dual focus in 
evaluating impact on students was implemented because, even if an intervention may not 
yield overall program level effects, its impact on individual students who improved as a 
result of the intervention can be examined to yield insights on “what works.” 



Demographic Characteristics of the Students 



The students ranged in age from 11 to 16; the mean age was 13.41. All students were in the 
8 th grade. The overall distribution of gender was 51.4% male and 48.6% female. A total of 
436 students on whom we had pre-tests completed the post-test survey and these students 



2 Pre-test and post-test are used interchangeably with pre-intervention and post-intervention throughout the 
report. 

3 The retention rate was not a product of dropping out of the study. Rather, it reflected temporary absence on the 
day the pre-test was administered and students moving to another community. 
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form the basis of the study group for the final data analysis. 4 The number of pre- and post- 
test surveys received from each school can be seen in Figure 1. 




□ Cleveland 
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□ Quincy 
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Figure 1: Distribution of Pre- and Post-test Surveys Received by School, N=436. 

The largest racial/ethnic group of surveyed students was Caucasian, followed by Asian/Asian 
Descent, Black/ African Descent, Hispanic, Other, and Biracial/Multi-ethnic (see Figure 2). 
The responses for Biracial/Multi-ethnic and Other included Chinese/Italian, African 
American/Asian, African American/Dutch/Italian, African American/Native American, 
African American/Caucasian, and Greek/Puerto Rican among others. It should be 
remembered that the racial/ethnic distribution is reflective of the schools from which the 
study group is drawn. 




□ Caucasian/White 
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Figure 2: Racial/ Ethnic Distribution of Study Sample, N=435. 

Characteristics of the intervention classrooms. Teachers provided the information on 
whether there was ability tracking in their school whereby students are placed in different 
classes based on some performance criteria. The study sample was almost equally divided 
between classrooms which did and did not track students (46.8% compared to 53.2%). These 
differences reflect school and/or school district policies. 



4 For the purpose of analysis, one group is omitted from final study sample. The “Experimental Lego Group,” 
which included 8 students on whom we had pre- and post-tests was omitted since they received a program with 
some variation from other experimental students in that school. 
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Discipline issues in the classroom were elicited either through self-reports by the classroom 
teacher or reported by the STEM Team coordinator, who had direct classroom experience or 
observed the classroom during survey administration. Discipline was seen as a problem in 
only 17% of the classrooms. 

Academic level of classrooms was also estimated by classroom teachers and/or coordinators 
in consultation with classroom teachers. Class level ranged from Honors (AP class level) and 
Advanced (College Prep), Mid-level (basic level math & science), Standard (lowest level at 
the school) and Mixed (academic level in math and reading covered a wide range of skill and 
grade levels) (see Figure 3). It should be remembered that different school systems use 
different rankings and this information should be viewed as suggestive rather than definitive. 




n Honors 
® Advanced 

■ M id-Level 
n Standard 

■ M ixed 



Figure 3: Class Level 

Student Questionnaire 

In addition to three attitude scales to be described later, the student questionnaire contained 
questions on educational goals, interest in a variety of careers related to STEM fields, and 
whether they knew any engineers. The post-test was identical to the pre-test and included an 
additional question on how interesting they found the activities on engineering in their 
science class and why. A copy of the Student Questionnaire can be found in Appendix B. 



Student Characteristics 

Future education goals. In response to the question, “Do you plan to go to college?” students 
could respond, “yes,” “no” or “not sure.” There was no statistically significant difference 
between pre- and post-test. At pre-test 91.0% replied affirmatively; at post-test 89.4% did so. 
The “not sure students” were 7.8% and 8.0% at pre-test and post-test, respectively. 
Differences by gender were not statistically significant at pre-test but almost reached 
significance at post-test (p< .07). At both times girls were more likely than boys to respond 
they planned to go to college (pre-test 92.9% compared to 89.3%; post-test 91.8% compared 
to 87.2%). 

Knowing engineers. At both pre-test and post-test students were asked if they knew any 
engineers. At pre-test 35.7% indicated that they knew engineers. Given the classroom 
exposure to engineers over the school year, the proportion indicating that they knew 
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engineers rose to 55.3% by post-test. The “not sure” group dropped from 34.3% to 24.7% 
between test administrations. This was a statistically significant pre-post difference (p<.001). 
Although not statistically significant, a slightly greater proportion of boys (36.8%) indicated 
they knew engineers at pre-test compared to girls (34.6%). By post-test, these proportions 
were reversed with 58.2% of girls indicating they knew engineers compared to 52.5% of 
boys, approaching statistical significance at pc. 07. This finding suggests that exposure to 
female role models was somewhat more memorable to girls than to boys. 

How interesting did students find the activities on engineering in this class. This question 
appeared on the post-intervention survey only. 5 A total of 411 students responded. Only 8.0% 
indicated that the activities were not at all interesting; 57.2% that activities were somewhat 
interesting and 34.8% found the activities very interesting. The average score was 2.27 
(between somewhat interesting and very interesting). There were no differences in average 
scores by gender. 

When asked to explain their answer, getting into the material, hands-on activities, specific 
modules, and the teams all were viewed as positive and even very positive aspects of their 
classroom experience. Among those who did not find the class at all interesting, many of the 
comments indicated that they found the class boring, redundant, and irrelevant because they 
were not interested in engineering, math, or science. 

Typical statements among those who found the class very or somewhat interesting included, 

The things that can be done with engineering are very interesting. I think it’s cool 
how we can make things out of everyday products that can be used all around the 
world. 

Because I love engineering...; building stuff out of nothing that will help everybody. 

One group of students who felt positively about the class used the word “fun” to explain their 
response. 

I couldn’t see myself as an engineer or even working with engineering, but after we 
did some of those projects in science, I realized that you could have a lot of fun with 
engineering, even though it wasn’t one of the jobs I wanted to work in the future. 

Working on all the projects was very fun and it made me think about what I am doing 
and how to do it best instead of just having only one possible way of doing it. 

Some of the students indicated that the class helped them learn or was educational, 
specifically mentioning the problem solving approach and hand-on activities used in the 
class. 



5 The question was scored as 1 =not at all interesting, 2=somewhat interesting , and 3=very interesting. It should 
be noted that students could provide as many explanatory responses as they wished. To get an estimate of the 
magnitude of the individual response category, the number of responses for each category was divided by the 
total number of valid responses. 
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It’s interesting because you can learn how different things work. 

[We use] many hands-on activities instead of reading out [of] the book. At the same 
time we learn a lot. 

Several students mentioned specific projects or activity modules as reasons for their positive 
evaluation of the class. 

I enjoyed it when we created objects like shoes because they used creativity and 
imagination, however, I did not enjoy experiments like the chocolate factory because 
the materials and steps involved limited my flexibility. For example, the lamp had to 
be a specific height above the factory. The thermometer had to be a specific 
temperature before you began using it as well. Also, I found the baseboards difficult 
to work with because the wires kept popping out. 

Because I learned new things through this program and to me it was fun to work with 
different problems such as the cast project, candy project and the bridge project. They 
were all fun to me. My favorite project was the candy one because you got to eat the 
candy afterward. 

Only 7% of respondents found the class boring and another 6% indicated that either they 
already had been exposed to similar activities in “kindergarten.” Some felt that while the 
activities “looked cool” they had problems understanding what the activities were about. 
Others found a few activities interesting, such as the solar cooker, but found others pointless. 

A few students mentioned role models, mentors, or teachers with particular reference to 
having the students and the engineers in the classroom. Several students noted that they 
enjoyed the teamwork and group work but were not sure they were interested in being an 
engineer. 

Several students felt that the class would be helpful for their future. 

I think that some engineering is an interesting profession to choose and would help 
me be successful and happy. Some topics in engineering, I’m not as interested. 

We learned new things and it taught us that most of the time there is more than just 
one solution to solve a problem. It was fun thing and I know it will help me with my 
life in the future. 

Attitudes toward Engineering, Math, and Science 

The attitude scales 6 were derived from several sources and were evaluated in a series of pilot 
tests for both reliability and construct validity. (Information on the reliability and validity of 
the attitude scales can be found in Appendix C.) 

6 Each attitude scales were scored as \=Strongly Disagree , 3=Not Sure, and 5 =Strongly Agree. 
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> The Attitude towards Math and Science Scales were adapted from Diane Doepken, 
Ellen Lawsky and Linda Padwa scales, which they had modified from the Fennema- 
Sherman Attitude Scales. The modification took place at the Gender Equity for 
Mathematics and Science Conference of Woodrow Wilson Leadership Program for 
Teachers in 1993. In our adaptation the original number of items pilot tested for the 
Attitude towards Math was 37 and for the Attitude towards Science was 27. Both 
scales were revised and successive versions were piloted in three communities. The 
final version for the 2003-2004 school year included 23 items for the Attitude 
towards Math scale and 21 items for the Attitude towards Science scale. Based on the 
2004-2005 pre-test further revisions of the two scales were undertaken resulting in a 
13-item Math scale and a 10-item Science scale. Both scales had Cronbach’s alpha 
internal consistency reliabilities above .85. 

> The Attitude towards Engineering Scale was adapted from the Survey of Attitudes 
towards Engineering scale developed by Robinson, Fadali, Carr, and Maddux (1999). 
The original number of items we pilot tested included 12 engineering and 24 activity 
interests. Successive revisions were based on pilot testing in four communities. The 
final 2003-2004 school year version contained 18 items. The scale was further 
reduced subsequent to the analysis of the pre-test data and resulted in a 12-item scale 
with Cronbach’s alpha internal consistency reliabilities above .80. 



Program Level Effect of Intervention 



Comparison ofpre- and post-inten>ention average attitude scores. 

Table 1: Simple Comparison of Pre- and Post-Intervention Average Attitude Scores, 
Paired t-tests 





Pre- 




Post- 






Significance 




Average 


s.d. 


Average 


s.d. 


Paired Mest a 


(2-tailed) 


Engineering 
Attitudes Scores 


3.24 


.723 


3.15 


.735 


3.235 


.001 


Math Attitudes 
Scores 


3.77 


.790 


3.76 


.801 


.492 


.623 


Science Attitudes 
Scores 


3.44 


.652 


3.53 


.756 


-3.105 


.001 



“Positive or negative sign of the t- test statistics reflect the difference between pre-test scores 
from the post-test scores. A positive t statistic shows that pre-test scores were higher. 

A simple comparison of pre- and post-intervention scores using paired t-tests found a 
statistically significant difference between pre- and post-intervention scores for engineering 
and science. The engineering score was lower at post-test while the post-test science score 
was higher, and the attitudes toward math score did not change. Table 1 summarizes these 
findings. 



Comparisons across schools. When these same scores were examined by school, at 
Ferryway, Forest Grove and Morris, post-test engineering attitude scores were statistically 
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significantly lower (pc. 05) as was the post-test math score at Quincy (pc. 05). The only 
school in which the engineering attitude score increased between pre- and post-test was 
Salemwood, although it was not statistically significant. Even though math attitude scores did 
increase positively between pre- and post-tests at three schools — Devotion, Ferryway and 
Forrest Grove — the increases did not reach statistical significance. Science attitude scores 
increased significantly at two of these three schools between test administrations (Devotion, 
pc. 01 and Forrest Grove, pc. 05) and almost reached significance at Ferry way (pc. 09). At the 
Morris and Quincy schools science attitude scores also increased between pre- and post-tests 
but did not reach significance. 

Comparison of average pre- and post-intervention attitude scores by gender 

In Table 2 we present the comparison of average pre- and post- intervention survey scores by 
gender. Boys’ average scores began higher at the pre-intervention survey on all three scales 
and continued to be statistically significantly higher at post-intervention survey. The overall 
trends noted in Table 1 persisted when the average attitude scale scores were examined by 
gender. 

Table 2. Simple Comparison of Average Pre- and Post- Intervention Attitude Scores by 
Gender 





Girls’ 




Boys’ 








Mean 


s.d. 


Mean 


s.d. 


E-test 


Pre-Intervention 
Engineering 
Attitudes Scores 


3.02 


.658 


3.43 


.727 


38.597*** 


Post-Intervention 
Engineering 
Attitudes Scores 


2.95 


.721 


3.35 


.691 


35.175*** 


Pre- Intervention 
Math Attitudes 
Scores 


3.63 


.817 


3.90 


.742 


13.338** 


Post- Intervention 
Math Attitudes 
Scores 


3.62 


.847 


3.90 


.728 


13.244*** 


Pre-Intervention 
Science Attitudes 
Scores 


3.34 


.641 


3.52 


.654 


8.523** 


Post-Intervention 
Science Attitudes 
Scores 


3.45 


.782 


3.60 


.725 


4.204* 



pc. 05; ** pc.01; *** pc. 001 



Both boys and girls average scores on engineering attitudes decreased between 
administrations and remained essentially unchanged on the math attitudes. Science attitudes 
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scores, on the other hand, increased for both genders between administrations with girls’ 
scores increasing by a slightly greater amount than boys’ scores (.11 and .08, respectively). 

Attitudes toward STEM professions/occupations. On both the pre- and post-test students were 
asked to indicate their possible interest in a list of professions/occupations that use 
knowledge about math, science and/or engineering. 7 Students rated only one profession, 
Computer Technician, significantly differently (p<.001) between pre- and post-test but in a 
negative direction; the pre-test score was significantly higher than the score at post-test. 

While not reaching significance, post-test scores increased slightly for Nurse, Electrician, and 
Chemist but remained unchanged for Engineer. 

When the professions/occupations were analyzed by gender, statistically significant 
differences were noted for Engineer (pc. 05) and an almost significant difference was noted 
for Chemist (pc.08). In both professions, girls’ scores increased between administrations by a 
small amount, while boys’ scores decreased for each of these professions. Overall boys’ 
scores increased between administrations for only two professions/occupations: Nurse and 
Electrician. Girls’ scores increased and by a larger amount for Nurse and Electrician and also 
increased positively for Chemist, Engineer, and Biologist. 



Discussion of Overall Program Effects Averaged over All Students 

The results presented up to this point do not provide support for the intervention’s impact on 
increasing girls’ attitudes toward STEM fields. In terms of attitudes toward STEM careers , 
however, relative to boys, girls registered more positive attitudes toward STEM careers in 
engineering, chemistry, and biology. In the following section we discuss the impact of the 
intervention on individual students. 

The fact that the attitudes toward science scores showed a significant increase over the course 
of the academic year speaks to the overall efficacy of the curriculum, which was presented in 
science classrooms. On the whole, students indicated that they found the classroom activities 
somewhat or very interesting and this was true even for those students who were not 
particularly interested in pursuing careers in math, science, or engineering. 

Several explanations suggest themselves to explain these limited findings. First, it may well 
be that the use of attitudinal scores may not adequately reflect the actual learning that has 
taken place in math, science, and engineering and that testing actual knowledge may result in 
more favorable findings. Another explanation is that for some students, a less favorable 
attitude toward engineering after the intervention reflects a more realistic appraisal of their 
thinking about engineering after they became better acquainted with it through the 
intervention. Indeed, the individual level outcomes bear this out: some students’ attitudes 
toward engineering were highly influenced by the intervention. 



7 The four-point scale was ranked: 1= Not At All Interested , 2=Not So Interested , 3 =Somewhat Interested , and 
4=Very Interested. The list of professions/occupations included: Physicist, Nurse, Auto Mechanic, Medical 
Doctor, Electrician, Computer Technician, Welder, Chemist, Engineer, Accountant, Biologist, and Architect. 
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Perhaps a more important indicator of the positive nature of the classroom experience can be 
found in the statistically significant increases between administrations in girls’ scores on two 
possible future careers, namely engineer and chemist. This finding stands in contrast to 
decreases in boys’ attitudes toward both of these professions between pre- and post- 
intervention survey administrations. 

Impact of School/T eacher on Program Outcomes 

It is important for any evaluation to examine if unique and unmeasured qualities of schools 
or teachers make a difference in an intervention’s outcome. These qualities can include the 
socioeconomic status of the school neighborhood, whether the school is in an urban or 
suburban community, teachers’ age, teachers’ educational level, years of service as a teacher. 
Eight middle schools participated in the implementation of the STEM Team strategy. In all 
but once case, only one teacher participated in the program from each school - one of the 
eight schools had three participating teachers. Because teachers and schools overlapped in 
most cases, any potential effect of the school is confounded by teacher effects. For that 
reason we refer to a joint school/teacher effect when comparing outcomes by school. 

An examination of change between the pre- and post-intervention scores (using the one-way 
analysis of variance procedure) indicated that among boys there were no statistically 
significant differences across the eight schools. Only on pre-post difference scores on 
attitudes toward science and becoming a chemist were there school/teacher differences that 
approached statistical significance (F = 1.895, df= 6, 207; p = .09 and F = 1, df= 6, 205, p = 
.07, respectively). For girls, however, which school they attended (the STEM team and/or 
teacher they had) made a difference in their attitudes toward engineering as a field of study 
(F = 2.169, df= 6, 202, p = .05), mathematics as a field of study (F = 3.197, df= 6,203, p = 
.01), and attitudes toward becoming an engineer (F = 2.161, df= 6, 192, p = .01). These 
results suggest that girls were more likely to be influenced by contextual factors in the school 
or by their science teacher, especially regarding engineering both as a field and as a career. 
These results pertaining to the school and teacher context need to be viewed as tentative 
because of the small number of schools involved in the evaluation. 



Impact of Intervention on Individual Students 

Whether or not there were gains (e.g., in attitudes toward science) or losses (e.g., in attitudes 
toward engineering) at the program level, there were a large number of individual students 
whose scores increased - that is, their attitudes toward STEM fields or careers became more 
positive - between pre-and post-test. We examined the impact of the intervention on 
individual students using regression analysis which yields information on which variables 
best predict whether a student’s attitudes will become more or less positive over the course of 
the intervention. 
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Impact on attitudes toward STEM fields 

The impact of the intervention on individual students’ attitudes toward STEM fields was 
analyzed in terms of difference scores. The difference scores capture the change in attitudes 
from the pre-test to the post-test, calculated by subtracting the pre-test scores from the post- 
test scores. Because a change from a score of 1 on the pre-test to 3 on the post-test is 
identical to a change from 3 to 5, the regression analysis included the pre-test score to control 
for each student’s attitudes before the intervention. In this way we were able to examine if 
students who started out with less positive attitudes toward STEM fields or careers changed 
more, the same as, or less than students who already had relatively more positive attitudes 
before the intervention began. 

In Table 3 the results of the regression analyses examining the difference scores on attitudes 
toward the STEM fields among male students is presented. The regression results are 

o 

expressed in beta scores. The predictor variables in this analysis are whether the student 
found the class interesting, the pre-intervention attitudes toward STEM fields, and the pre- 
post difference scores on attitudes toward STEM fields. Other predictors such as age, 
knowing engineers, and self-assessment of proficiency in the STEM subjects were examined 
but eliminated from the final regression analysis because they did not make a significant 
contribution to understanding change in attitudes from pre- to post-intervention for a given 
STEM field. 

The results in Table 3 show that for each pre -post difference score in attitudes toward 
engineering, math, and science the most powerful predictor is a student’s pre-intervention 
attitude toward the same STEM field before the project was implemented. The negative sign 
of the beta coefficient suggests that students who started out with more positive attitude 
scores ended up not changing much and students with less positive pre-intervention attitude 
scores had large difference scores, indicating greater change from the pre-test to the post-test. 

There are two possible explanations for this finding. The first and the most optimistic is that 
students who initially had lower positive attitude toward a STEM field changed the most. 

The second interpretation is that the negative beta coefficient is a statistical artifact because 
students who initially had more positive scores did not change much because they were 
already positive. In methodological terms this second explanation would be referred to as a 
ceiling effect. Ceiling effects are considered to be an artifact of measurement, not substantive 
findings related to the impact of an intervention. We believe that the most likely 
interpretation is that the intervention was more successful with the students who did not have 
positive attitudes initially but that this relationship is not as strong as the beta coefficients 
would indicate. 



8 Beta scores reflect the strength of the relationship between a given predictor such as 
Interesting Class and the outcome variable, for example, Pre-Post Difference in Attitudes 
toward Engineering, in the context of all other predictor variables in the analysis. 
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Table 3. Regression Analyses Predicting Difference Scores in Attitudes toward STEM Fields 
among Boys, Expressed in Standardized Regression Coefficient, beta. 



V=195 

Predictor Variables 


Pre-Post Difference 
In Attitudes toward 
Engineering 
beta 


Pre-Post Difference 
In Attitudes toward 
Math 
beta 


Pre- Post Difference 
In Attitudes toward 
Science 
beta 


Interesting class 


.139* 


-.007 


.141* 


Pre Attitude toward 
Engineering 


- 475 *** 


. 220 * 


.051 


Pre Attitude toward 
Math 


.155* 


- 497 *** 


3 ^ 9 *** 


Pre Attitude toward 
Science 


.011 


.049 


-. 454 *** 


Pre -post Difference 
in Engineering 


- 


238 *** 


.139 


Pre -post Difference 
In Math 


.227** 


- 


.138 


Pre -post Difference 
In Science 


.137 


.143 


- 


Variance Explained 

(R 2 ) 


27% 


24% 


27% 



*p<.05; ** pc. 01; ***p<.001 



It is of interest to note that among boys a positive pre-post difference score in attitudes 
toward engineering is additionally predicted by reports of finding the intervention classes 
interesting, having a positive attitude toward mathematics before the intervention, and 
coming to have more positive attitudes toward math after the intervention. These variables 
together explain 27% of the observed differences in male students’ pre-post difference scores 
in attitudes toward engineering. A similar amount of variance is explained in the observed 
differences in male students’ pre-post difference scores in attitudes toward math and science 
(24% and 27%, respectively). Finding the class interesting makes a difference for science but 
not for math. Having an initial positive attitude toward math before the intervention 
contributes to predicting the pre-post difference in attitudes toward science but an initial 
positive attitude toward science does not appear to be important for a larger positive 
difference score on math attitudes. This finding that an initial positive attitude toward 
mathematics making 8 th grade boys more open to developing positive attitudes toward 
science but not the other way around suggests that mathematics is the more basic of the 
STEM fields. If you like math, you’ll like science after the intervention, but if you like 
science initially, that initial positive attitude is not likely to generalize to liking math later. 



23 



Table 4. Regression Analyses Predicting Difference Scores in Attitudes toward STEM Fields 
among Girls, Expressed in Standardized Regression Coefficient, beta. 



N=\91 

Predictor Variables 


Pre-Post Difference 
In Attitudes toward 
Engineering 
beta 


Pre- Post Difference 
In Attitudes toward 
Math 
beta 


Pre- Post Difference 
In Attitudes toward 
Science 
beta 


Interesting class 


3^2*** 


.002 


.105 


Pre Attitude toward 
Engineering 


-567*** 


.223* 


287*** 


Pre Attitude toward 
Math 


.203* 


-.384*** 


-.090 


Pre Attitude toward 
Science 


.109 


-.059 


- 294*** 


Pre -post Difference 
in Engineering 


- 


289*** 


.330*** 


Pre -post Difference 
In Math 


214** 


- 


.057 


Pre -post Difference 
In Science 


245*** 


.057 


- 


Variance Explained 

(R 2 ) 


41% 


21% 


20% 



*p<.05; **p<.01; *** pc.001 



Predicting the girls’ change in attitudes toward STEM careers from before to after the 
intervention follows a similar pattern of beta coefficients, with the initial attitude scores 
showing the largest but negative effects (see Table 4). Finding the intervention classes 
interesting appears to make a large difference in developing more positive attitudes toward 
engineering after the intervention but does not change the attitudes toward math or science. 
The most remarkable finding in the girls’ results is that the predictor variables in the 
regression analysis explains fully 41% of the observed differences in female students’ pre- 
post difference scores in attitudes toward engineering. The variance explained for difference 
scores in math and science is half of that. We interpret this finding to mean that the 
intervention had a more powerful effect on girls’ attitudes toward engineering than it did on 
their attitudes toward math and science. Moreover, the intervention had a more powerful 
impact regarding changes in girls’ than boys’ attitudes toward engineering. 

In Tables 5 and 6 the results of the regression analyses predicting changes in students’ 
interest in STEM careers is presented. For both boys (Table 5) and girls (Table 6) the most 
influential predictor is having a less positive attitude toward the career in the pre-test, as 
indicated by high but negative beta coefficients. We believe that similar to the difference 
scores in attitudes toward STEM fields, this pattern of results is brought about by students 
starting out with low scores benefiting more from the intervention. Again, however, the 
impact of the intervention is not likely to be as strong as the magnitude of the beta 
coefficients would suggest because of the possibility of a ceiling effect. 
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Table 5. Regression Analyses Predicting Difference Scores in Attitudes toward STEM 



Careers among Boys, 


expressed in Standarc 


ized Regression Coefl 


'icient, beta. 


V=189 

Predictor Variables 


Pre-Post Difference 
In Engineer 
beta 


Pre-Post Difference 
In Chemist 
beta 


Pre-Post Difference 
In Biologist 
beta 


Interesting class 


230*** 


.052 


-.025 


Pre-post Difference 
in Engineering 


.196* 


.065 


.031 


Pre-post Difference 
In Math 


.140* 


-.065 


-.064 


Pre-post Difference 
In Science 


-.108 


.145* 


.261*** 


Pre-Engineer 


-.495*** 






Pre-Chemist 




- 499 *** 




Pre-Biologist 






- 403*** 


Variance Explained 

(R 2 ) 


32% 


28% 


28% 


* pc. 05; ** pc.01; *** pc.001 

Table 6 . Regression Analyses Predicting Dil 
Careers among Girls, Expressed in Standarc 


Terence Scores in Atti 
ized Regression Coef 


tudes toward STEM 
’icient, beta. 


N=m 

Predictor Variables 


Pre-Post Difference 
In Engineer 
beta 


Pre-Post Difference 
In Chemist 
beta 


Pre-Post Difference 
In Biologist 
beta 


Interesting class 


.150* 


.038 


.019 


Pre-post Difference 
in Engineering 


231*** 


.031 


.009 


Pre-post Difference 
In Math 


.009 


.012 




Pre-post Difference 
In Science 


233*** 


.303*** 


.153* 


Pre-Engineer 


- 561*** 






Pre-Chemist 




-.450*** 




Pre-Biologist 






- 482*** 


Variance Explained 

(R 2) 


41% 


26% 


27% 



*pc.05; ** pc. 01; *** pc.001 



In comparing variance explained in Tables 5 and 6 we see that similar to the finding 
regarding girls being more influenced by the intervention in their attitudes toward 
engineering as a field, they are more influenced toward engineering as a career. This is 
evidenced by the 41% of the observed differences in female students’ pre-post difference 
scores in interest in becoming an engineer. 
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We interpreted the findings that the intervention had the most influence on girls’ attitudes 
toward engineering as a field and engineering as a career in terms of girls’ openness to new 
information on engineering. We believe that this openness is due to having less information 
on engineering before the intervention relative to boys rather than to a general openness. 
Indeed, girls’ interests were influenced at the same level as boys’ on biology and chemistry 
as fields of study and careers, which are better known fields. We believe that the intervention 
was able to influence girls to a greater extent because girls had fewer preconceived notions 
about engineering. This interpretation is bolstered by the finding that before the intervention 
slightly fewer girls reported knowing engineers than did boys. 

Another finding that showed girls to be influenced more by the STEM Team strategy than 
boys can be seen in the comparison of the impact of school/teacher as a context for the 
impact of the intervention . 9 While boys’ changes from the pre- to post-testing did not differ 
across schools or teachers, the comparisons for girls showed statistically significant changes, 
especially in attitudes toward engineering as a field of study and a career and also 
mathematics as a field of study. The hypothesis of school/teacher context having more of an 
influence on how girls respond to the intervention is open to interpretation. It could be a 
function of their lower levels of exposure to engineering, hence heightened sensitivity, as 
suggested above. It could also be that girls were more reactive to the conditions under which 
they were exposed to the intervention. Future research is needed to examine teacher and 
school effects separately using a larger sample. 



Evaluation Summary 



Teacher Outcomes 

Teachers found the intervention highly valuable both for themselves and their students. For 
themselves one key element in the success of the undertaking is in developing a clearer 
understanding of engineering concepts, gender equity, and the expertise contributed by each 
STEM Team member to the classroom. Regarding STEM Team members’ contributions, 
teachers reported that engineers provided their expertise in science; college faculty, graduate 
and undergraduate students brought in the most current research findings; the industry 
representative in providing real-life, fresh perspectives from their field experience, and the 
middle school teachers provided their knowledge and understanding of how to approach and 
teach young adolescents. Teachers reported that they benefited from having access to a 
curriculum that was not only of high quality but had built into it gender equitable practices 
and principles. 

For their students, the benefits teachers highlighted were role models, glimpses into the real 
world of work for engineers, and a curriculum that made it possible for all students to 
achieve up to a benchmark of proficiency rather than a competitive classroom environment 
where only a few students are engaged in the projects. 

9 We refer to the joint effect of school and teacher because in seven out of eight schools only 
one teacher participated in the program making it impossible to tease out school effects from 
teacher effects. 
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STEM Team outcomes 

Each STEM Team was composed of a coordinator, teachers, faculty members, undergraduate 
and graduate students, and industry representatives. STEM Team members reported they 
enjoyed the regular meetings that served to keep the teams focused and remarked on the 
synergy generated by the diversity in team membership. The study showed that it is feasible 
to bring together individuals from different domains to work together toward enhancing the 
teaching of STEM fields, especially the new engineering strand. 

Student Impact 

Program level outcome. A simple comparison of pre- and post- intervention scores using 
paired t-tests found a statistically significant difference between pre-and post-intervention 
attitudes toward engineering and science as a field of study. Attitudes toward science became 
more favorable after the intervention than it was before. However, the attitudes toward 
engineering were less positive at post-test. Attitudes toward mathematics did not change. 
Analyses by gender showed that overall, girls’ attitudes showed a greater increase than boys’ 
attitudes in becoming engineers, biologists, or chemists after the intervention. 

Individual level outcomes. In spite of the fact that students’ overall scores on attitudes did not 
become more positive after the intervention, there were a large number of individual students 
whose scores increased — that is, their attitudes toward STEM fields or careers became more 
positive — after the intervention. We examined the impact of the intervention on individual 
students using regression analyses which yield information on what variables best predict 
whether a student’s scores will increase or decrease. The results showed that students who 
started out with lower positive attitudes toward both STEM fields and STEM careers made 
more gains after the intervention than students who started out with more positive attitudes. 
Moreover, the intervention had a greater impact on female students’ attitudes toward 
choosing engineering as a career than it did on male students’ attitudes. 



Conclusions 

The evaluation shows that the 4 Schools for WIE project has had the intended effect on: 

• Capacity building among 8 th grade science teachers’ confidence to teach engineering; 

• Program level increase in attitudes toward science as a field of study but not in 
engineering or mathematics; 

• While in absolute numbers more boys than girls reported they would consider 
engineering, biology, and chemistry as a career, the change in girls’ interest in these 
careers was greater than the change in boys’ interested in these careers after the 
intervention; 

• Individual level impact of the intervention on girls’ attitudes toward engineering both as a 
field of study and as a career. 
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The results show that in general, among both girls and boys, students who show the most 
positive change relative to where they started out in the beginning of the academic year are 
those who had relatively less positive attitudes toward STEM fields and careers. While some 
of this effect may be an artifact of a ceiling effect, this finding suggests that the intervention 
has the potential to benefit those in most need of help instead of merely making students who 
already hold positive attitudes hold even more positive views toward STEM fields and 
careers. 

A shortcoming of the evaluation is that it did not allow for following up the students who 
appeared to benefit from the intervention. We cannot answer the question what happens to 
these students as they continue on to high school. Do they, in fact, take more courses in math, 
science, and technology and at higher levels (advanced placement)? How do their ideas 
regarding potential careers change over time? Without the ability to follow these young 
people beyond 8 th grade, we cannot assess the full impact of the 4 Schools for WIE 
curriculum. In addition, without true control groups using random assignment, we cannot 
definitively attribute change to the efficacy of the curriculum. A further weakness of the 
present study is that it did not allow for an estimation of the school and teacher effects as 
separate contexts for the intervention. This is a particularly important point to examine in 
future studies because girls appeared to be more influenced by the school/teacher context 
regarding engineering than did boys. 

What we do know, is that for the teachers the exposure to expertise from “real” engineers (as 
one teacher put it) — professors, graduate and undergraduate students and industry 
representatives coupled with the training that was such an integral part of this intervention — 
made a significant difference in both their understanding of engineering and how to teach this 
subject effectively. That may in fact be one of the most “significant” impacts of the project 
over time. 

All in all, the STEM team intervention succeeded in its goal to improve teachers’ capacity to 
teach the new engineering strand in the 8 th grade science curriculum that has been 
incorporated into the Massachusetts Science Frameworks. It succeeded partially in improving 
all students’ attitudes toward science as a field of study but not engineering or mathematics. 

It was more effective in fostering positive attitudes toward STEM fields and careers among 
male and female students who started out with less positive attitudes. Perhaps even more 
significantly, it resulted in more girls coming to view engineering and chemistry as possible 
careers for themselves and had a larger impact on girls than it did on boys. 
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Appendix A 

Teacher Survey and Report 



Teacher Feedback form 

4 Schools for WIE Report on Teacher Surveys, June 2004 
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Teacher Feedback Form 



School: Teacher: STEM Team: 

The following questions concern your experience in implementing the 4 Schools for WIE 
activities in your classrooms during the 2003-2004 academic year. Thank you in advance for 
completing this survey, it will help us in planning for next year. You may fill this out on line 
and return it to: fmarx@wellesley.edu or to: Fern Marx, WCW, Wellesley College, 106 
Central St., Wellesley, MA 02481. 



1 . The following are the five goals and anticipated outcomes for all 4 Schools for WIE 
Team Projects this year. 

Please rate how well you think the goals were met from “1= did not meet this goal at all” 
to “5=met this goal very successfully.” Circle one number in each row. If you feel that 
the goal was only partially met, please explain your answer in a sentence below the goal. 



a. Teach concepts about the practice 
and uses of engineering. 



did not meet this 
goal at all 

1 



met this goal 
very successfully 



Please explain: 



b. Integrate engineering design 
process into classroom activities 



3 4 



Please explain: 



c. Apply gender-inclusive approaches 
in the classroom. 



3 4 



Please explain: 



d. Communicate the connection 

between classroom activities and 1 2 3 4 5 

the Massachusetts state frameworks 



Please explain: 

e. Capitalize on school or district 
resources and local areas of interest 



2 3 4 5 



Please explain: 
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2. To what extent do you feel that the training and support available to you from your 
STEM Team adequately prepared you for teaching the 4 Schools for WIE curriculum 
units and activities? Please circle one answer below and explain your response below. 

a. I felt the preparation/training from my STEM Team was: 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



b. I felt the support available from my STEM Team was: 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



3. What, if any, additional training or support would you have liked to receive from 
your STEM Team? 



4. How effective do you believe the curriculum and activities were in preparing 
students for the engineering strand of the MCAS exam? 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



5. What suggestions do you have for making the curriculum more effective for the next 
implementation? 
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6. A major part of the 4 Schools for Women in Engineering Project is the idea of 

providing STEM Teams consisting of practicing engineers in industry and academia, 
as well as engineering graduate and undergraduate students. 

a. In your opinion, how effective was having an all female STEM Team in providing 
role models for girls in science and engineering? Please explain your answer with 
specific examples, if possible. 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



b. How effective was an all female STEM team for boys ? Please explain your answer 
with specific examples, if possible. 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



c. Would the same results be possible if the visitors were only from companies, or 
only professors or only engineering students? Please explain your answer. 



7. Thinking back over your experience this academic year with the 4 Schools for 
WIE, what stands out for you as the most significant experience(s)? 



8. How effective do you feel the curriculum was in stimulating girls’ interest in 
STEM, in taking addition math/science courses and thinking about future careers 
related to STEM? Please explain your answer with specific examples, if possible? 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



9. Do you feel that having an all female STEM Team was successful in making the 
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math, science, and engineering curriculum more gender-neutral? Please explain your 
answer with specific examples, if possible? 

Very Inadequate Poor Fair Good Excellent 
1 2 3 4 5 

Please explain: 



10. Do you have any additional thoughts about this year’s experience that you would like 
to share with us? 



Thanks for your help! 
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4 Schools for WIE 
Report on Teacher Surveys 
June 2004 

The findings from the 2004 Teacher Feedback surveys for the 4 Schools for Women 
in Engineering (WIE) program are discussed in the following report. Six teachers responded 
to the survey, two of whom taught at Forest Grove Middle School and one each taught at 
Cleveland, Ferryway, Josiah Quincy, and Salemwood Schools. The three teams represented 
include Northeastern, Tufts, and WPI. 

Five Goals 

Four Schools for WIE established five goals and anticipated outcomes for all of their 
programs this year. Results presented in Table 1 show that these goals were met to a large 
extent (although according to one respondent, with the exception of applying gender 
inclusive approaches in the classroom, the goals were rarely met). Overall the most 
successfully met goal was applying gender inclusive approaches in the classroom. Teachers 
wrote that they encouraged students to try taking a non-traditional gender role; one wrote that 
she “mixed groups up according to gender being careful to encourage girls and boys to try 
different roles.” Others suggested that “success is due to awareness of gender equity goals”; 
one teacher wrote “I always feel the classroom was gender-inclusive but because of this 
initiative, I have evaluated my lesson plans more carefully especially when teaching simple 
machines and work/force concepts.” 

The goals of teaching concepts about the practice and uses of engineering and 
communicating the connection between classroom activities and the MCAS frameworks 
were met fairly successfully or very successfully. Regarding teaching the concepts of 
engineering, one teacher wrote, “Students had a very clear idea of the engineering steps 
involved by the end,” and another, “I felt we met this goal by teaching through concepts and 
presentation/activities.” As far as communicating the connection between classroom work 
and the state frameworks, teachers wrote that “students are very clear why they were learning 
engineering and how it connects to the frameworks/MCAS,” and “we actually sat down and 
compared each goal to make sure it fit [within the framework] !” 

The two goals with the lowest overall ratings were integrating engineering design 
processes into classroom activities and capitalizing on school or district resources and local 
areas of interest. One teacher wrote, “the process of integrating design engineering into 
classroom activities was evident during the planned activities, but I felt time constrained, 
personally, from further implementing more of the engineering design process.” Another 
teacher commented that the team’s orange juice project discovered the essence of integrating 
the engineering design process into the classroom activities, writing “the OJ module was just 
that!” Another teacher wrote, “I did this on my own with other activities, but not in 
connection with my STEM Team.” In terms of capitalizing on school or district resources 
and local areas of interest, one teacher felt it did not happen at all. Another wrote, “I think 
that we may need to work a little bit on this goal.” 
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Table 1: Meeting the Five Goals (n=6) 



Goal 


Mean 

Score 10 


Teach concepts about the practice 
& uses of engineering 


4.33 


Integrate engineering design 
process into classroom activities 


3.83 


Apply gender- inclusive approaches 
in the classroom 


4.83 


Communicate connection between 
classroom activities & MA state 
frameworks 


4.17 


Capitalize on school or district 
resources and local areas of interest 


3.67 



STEM Teams 

The teachers were asked to rate the STEM Teams in a number of areas. Overall, the STEM 
Teams seem to have done an exceptional job (See Table 2). All six respondents rated the 
effectiveness of having an all-female STEM Team for both girls and boys as excellent. One 
teacher wrote, “The students worked well with the practicing engineers, instructors, and 
students,” and another, “girls felt more comfortable participating and asking questions.” As 
far as its effectiveness on the boys, one wrote, “All students were engaged in doing and 
understanding when the engineering student was in the classroom. I really don’t think they 
gave a lot of thought to the fact that the student was female,” and another wrote, “male 
students involved themselves in the activities without regard to the gender of presenter. The 
male students were also used to having female teachers. It might be interesting to have an all 
female STEM Team go into classrooms where males were the teachers.” Five out of the six 
teachers rated the teacher preparation and training and the available support provided by the 
STEM Teams as excellent. One teacher, reflecting the sentiment of the majority of the 
group’s responses, wrote, 



10 The responses were scored from 1 =”did not meet goal at all”, 3=”met goal somewhat”, to 5=”met goal very 
successfully”. 
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I stand completely amazed at the team support of time and energy which the team 
members put into designing the implementing the lessons which we developed. If 
teachers had the resources to invest in each lesson as we did the engineering lessons, 
what a wonderful world it would be. I am thankful for their unending support. 

Another wrote that, “They were always willing to accommodate our schedules and come in 
to the class on a drop of a dime.” Only one teacher felt that more time could have been spent 
in preparing for implementation. All six of the respondents indicated that the effectiveness of 
making the STEM curriculum more gender neutral was either good or excellent; “I think they 
understand that it is an easy option for males and [we are] exposing females to it.” 

Table 2: Regarding the STEM Team (n=6) 





Mean Score 11 


Teacher Preparation/Training 


4.50 


Available Support 


4.83 


Effectiveness of All-Female 
Team for Girls 


5.00 


Effectiveness of All-Female 
Team for Boys 


5.00 


Effectiveness of Making 
STEM More Gender-Neutral 


4.50 



Additional Training and Support: The teachers were asked for suggestions on what 
additional training and/or support they would like to have received from their STEM Team. 
One teacher wrote, 

I would like to work with WPI further on my own “knowledge” of the technology 
part of the science and technology MCAS frameworks - not only for content but also 
to work on creative ways to implement technology topics in the classroom. 

Another wrote, 

I’ve never had the advantage of classroom support from a Team member familiar 
with the project’s activities. In both 2003 and in 2004 I’ve attempted to guide my 
classes through certain activities either alone or with a STEM Team member who was 
just becoming familiar with the project. Most work with STEM Team member who 
knew the project was implemented in classes at the other school involved in the 
project. 

Two teachers responded that plenty of training and support were provided. Another wrote, 
“Who could ask for anything more???? I am a ‘happy camper’ and felt this was a better year 
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The responses were scored from l=”very inadequate”, 3=”fair” to 5=”excellent”. 
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than last. We were able to do more lessons and had better experiences with the ones that were 
developed last year.” 

Change in Teams?: Participants were asked whether the same results would be possible if the 
visitors were only from companies, only professors, or only engineering students. Four out of 
the six teachers responded, and they all thought the team setup was just right. One wrote, 

‘‘No - the ‘mix’ worked well as it provided support and guidance for teachers which led to 
‘smooth’ implementation.” Others wrote, “I think the mix of all ranges of people involved in 
engineering is beneficial. But of course my students enjoyed having college students working 
with them. It gave them a positive role model of someone in college,” “no - 1 think pulling 
from all fields is effective,” and 

I think that the combination added greatly to the preparation but I do not think the 
students are completely cognizant of each presenter’s actual role in the Team. They 
generalize that everyone was an engineer. We even use that general term in 
conversation with the students: “When the engineers return, we will have the 
presentations.” 

Effectiveness of Curriculum 

Respondents were asked to what extent the new curriculum affected the students. 
Results show that the curricula and activities were quite effective in preparing students for 
the MCAS exam (See Table 3). All of the participants marked this item as either good or 
excellent. One teacher wrote, “After having seen the 2004 MCAS for Science Engineering 
and Technology, I am fairly confident that the curriculum designed by the STEM Team does 
a good job in addressing the Engineering strand.” Another wrote, “My students actually said 
they knew that part of the MCAS best!” As far as stimulating girls’ interests in STEM, half 
of the respondents indicated the program was excellent and the others marked “good” or 
“fair.” Teachers’ responses included, “Exposure and comfort level when working with any 
topic builds confidence. Student confidence is an important factor in course selection,” “I 
think the curriculum stimulated girls’ interests, but I feel it was a beginning, and to overcome 
the girls’ initial refrain from these courses might take a longer initiative,” “very good in 
stimulating girl’s interest, not so sure about increasing their desire to take more math/science 
courses or about future careers,” and “many students have approached me to talk about their 
career paths and options for high school courses.” 

Table 3: Effectiveness of Curriculum & Activities (n=6) 





Mean Score 12 


Preparation for MCAS Exam 


4.33 


Stimulating Girls’ Interest in 
STEM 


4.33 
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The responses were scored from l=”very inadequate”, 3=”fair” to 5=”excellent”. 
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Suggestions for the Next Implementation: The teachers were asked for suggestions for 
making the curriculum more effective for the next implementation. Suggestions included: 

• More communication between all teachers involved in the project, preferably via the 
Online Community Board 

• A clearer sense of the time-line 

• More frequent communication regarding everything needed to be done in order to 
complete all expectations by given deadline 

• Change in the use of tools - “The MCAS test had many references to tools that are 
not used in my general science curriculum. In the activities we had designed in 
engineering, the tools are relatively simple and do not reflect the knowledge of tools 
that seems to be desired” 

Additional Feedback 

Significant Experiences: The teachers were asked to comment on what stood out for them as 
the most significant experience(s) during this past academic year with 4 Schools for WIE. 
One teacher wrote that her most significant experiences were “the support, content 
knowledge, and increase in teacher ‘comfort’ in working with the technology strand”; 
another wrote that “one of the best experiences was having college students come into the 
classroom” and another wrote that “taking students to Tufts to see first-hand what it is like to 
study engineering and meet the professors” was the most significant. Another wrote, 

The thing that stands out most for me is the difference between the expectations of 
the professional engineer involved versus the reality of how things actually play out 
in the classroom; the lack of flexibility of prerequisites for student understanding of 
certain ideas. Some engineers tend to have a fixed conception of how learning must 
proceed in order for understanding to occur. 

A significant experience for one teacher was that her team was able to cover a lot of 
material related to engineering. 

We were able to do three times the amount of engineering related activities as we 
accomplished the first year. I was most grateful for the bridge introduction we added. 
The STEM Team asked the teachers what we deemed more useful in meeting the 
MCAS framework requirements and we all agreed that we needed more on bridges. 
The Team then went to work and put together a very nice, lively presentation and 
activity on bridges which we all developed together and which I, as a teacher, 
personally needed to augment my knowledge. That proved to be very fruitful as there 
were many bridge questions on the MCAS tests. 

Additional Comments: A number of comments were offered when respondents were asked if 
there were any additional thoughts they would like to share about this year’s experience. One 
wrote, 



The ending of this year has left me with the confidence that I need to enjoy 
implementation next year, from pre-test to implementation to post-test. It has taken 
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me until now to actually see what really needs to happen in order for this to work as it 
is expected to unfold in the classroom. 

Another wrote, 

The first year of the program I was happy with the results. The second year was even 
better as we were able to revise and refine for even better results. I hope to keep this 
going even beyond year #3, but I would like to continue with WPI support. 

Another was delighted with the experience: “I enjoyed working with all the members of the 
Tufts staff and students,” and other offered their gratitude, “A big “Thank you” for 
everyone’s sincere efforts.” 
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Appendix B 
Student Survey 



STUDENT ASSENT FOR PARTICIPATION 



Today I would like to ask you to complete a short survey on attitudes towards mathematics, 
science and engineering, and careers in these fields. Your participation is voluntary. If you 
decide you don’t want to complete the survey at all or to stop at any time or to skip questions, 
you are free to do so. 

If you would like to complete this survey, please check “Yes” 

Yes No 



Your Name: 



Your School’s name: 



Your Teacher’s Name: 



Today’s Date: 



The following questions are about engineering. Please read this paragraph before you 

answer the questions. 
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Engineering is a career that uses math and science to invent new products and solve 
problems that improve everyone’s life. There are many different types of engineering, 
such as chemical, electrical, computer, mechanical, civil, environmental, biomedical, 
and design. The word engineer comes from the Greek word for imagine. Engineers or 
“ imagineers ” not only build bridges and cars but also invent new fabrics, foods, and 
even virtual reality amusement parks. In fact, “imagineers” work with almost 
everything we eat, drink, wear, touch, see, smell, and hear in our daily lives. 



On the following pages is a series of sentences. Please mark your answer sheets by indicating 
how you feel about them. Suppose a statement says: 





Strongly 


Somewhat 


Not 


Somewhat 


Strongly 


Example 1: 


Disagree 


Disagree 


Sure 


Agree 


Agree 




1 


2 


3 


4 


5 


I like engineering. 


O 


O 


O 


O 


O 



As you read the sentence, you will know whether you agree or disagree. If you strongly 
agree, put an X in the circle under Number 5. If you agree, but not so strongly, or you only 
"sort of" agree, put an X in the circle under 4. If you disagree with the sentence very much, 
put an X in the circle under 1 for strongly disagree. If you disagree, but not so strongly, put 
an X in the circle under 2. If you are not sure about a question or you can't answer it, put an 
X in the circle under 3. 

Even though some statements are very similar, please answer each statement. This is not 
timed; work fast, but carefully. 

There are no "right" or "wrong" answers. The only correct responses are those that are true 
for you. Whenever possible, let the things that have happened to you help you make a choice. 



Please fill in only one answer per question. 
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1. Attitudes toward Engineering 





Strongly 

Disagree 

1 


Somewhat 

Disagree 

2 


Not 

Sure 

3 


Somewha 

t 

Agree 

4 


Strongly 

Agree 

5 


1. A degree in engineering will allow 
me to obtain a well paying job 


O 


O 


O 


O 


O 


2. I am not interested in any career 
that uses math and science. 


O 


O 


O 


O 


O 


3. I like fixing broken appliances. 


o 


O 


o 


o 


o 


4. At the science museum, I like the 
exhibits on robotics. 


o 


O 


o 


o 


o 


5. A degree in engineering will allow 
me to obtain a job I like doing. 


o 


O 


o 


o 


o 


6. I have no interest in helping 
design the space station. 


o 


O 


o 


o 


o 


7. Engineering skills will allow me 
to better society. 


o 


o 


o 


o 


o 


8. A degree in engineering will give 
me the kind of lifestyle I want. 


o 


o 


o 


o 


o 


9. I am interested in designing 
better artificial limbs. 


o 


o 


o 


o 


o 


10. 1 would like to learn how to make 
safer cosmetics. 


o 


o 


o 


o 


o 


11. Engineering interests me because 
I like to think about solving 
technical problems. 


o 


o 


o 


o 


o 


12. 1 am not interested in what makes 
machines work. 


o 


o 


o 


o 


o 
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2. Attitudes toward Math 



Please use the same instructions you used for completing the Attitudes toward 
Engineering. 





Strongly 

Disagree 

1 


Somewhat 

Disagree 

2 


Not 

Sure 

3 


Somewhat 

Agree 

4 


Strongly 

Agree 

5 


1. Math is a worthwhile, necessary 
subject. 


O 


O 


O 


O 


O 


2. Math is not important for my life. 


O 


O 


O 


o 


O 


3. Math has been my worst subject. 


o 


O 


o 


o 


o 


4. I see math as something I won’t use 
very often when I get out of high 
school. 


o 


O 


o 


o 


o 


5. I would consider choosing a career 
that uses math. 


o 


O 


o 


o 


o 


6. I study math because I know how 
useful it is. 


o 


O 


o 


o 


o 


7. Math is hard for me. 


o 


O 


o 


o 


o 


8. I’ll need a good understanding of 
science for my future work. 


o 


O 


o 


o 


o 


9. I’m not the type to do well in math. 


o 


o 


o 


o 


o 


10. Most subjects I can handle OK, but I 
just can’t do a good job with math. 


o 


o 


o 


o 


o 


11. 1 am sure I could do advanced work 
in math. 


o 


o 


o 


o 


o 


12. 1 can get good grades in math. 


o 


o 


o 


o 


o 


13. I’m no good in math. 


o 


o 


o 


o 


o 
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3. Attitudes toward Science 



Please use the same instructions you used for completing the Attitudes toward Math. 





Strongly 

Disagree 

1 


Somewhat 

Disagree 

2 


Not 

Sure 

3 


Somewhat 

Agree 

4 


Strongly 

Agree 

5 


1. I am sure of myself when 
I do science. 


O 


O 


O 


O 


O 


2. I would consider a career 
in science. 


O 


O 


O 


O 


o 


3. I don’t expect to use much 
science when I get out of 
school. 


o 


O 


o 


o 


o 


4. Knowing science will help 
me earn a living. 


o 


O 


o 


o 


o 


5. I’ll need science for my 
future work. 


o 


o 


o 


o 


o 


6. I know I can do well in 
science. 


o 


o 


o 


o 


o 


7. Science will not be 
important to me in my 
life’s work. 


o 


o 


o 


o 


o 


8. Science is a worthwhile, 
necessary subject. 


o 


o 


o 


o 


o 


9. Most subjects I can 
handle OK, but I just 
can’t do a good job with 
science. 


o 


o 


o 


o 


o 


10. 1 am sure I could do 

advanced work in science. 


o 


o 


o 


o 


o 
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4. Your Future 



Here is list of jobs that use knowledge about math, science and/or engineering. How 
interested are you in these kinds of jobs? If you are Very Interested, put an X in the 
circle under Number 4. If you are Somewhat Interested, put an X in the circle under 3. If 
you are Not So Interested, put an X in the circle under 2. If you are Not At All Interested, 
put an X in the circle under 1. 

There are no “right” or “wrong” answers. The only correct responses are those that are 
true for you. 





Not At All 


Not So 


Somewhat 


Very 




Interested 


Interested 


Interested 


Interested 




1 


2 


3 


4 


1. Physicist 


O 


O 


O 


O 


2. Nurse 


O 


o 


O 


O 


3. Auto Mechanic 


o 


o 


o 


o 


4. Medical Doctor 


o 


o 


o 


o 


5. Electrician 


o 


o 


o 


o 


6. Computer 
Technician 


o 


o 


o 


o 


7. Welder 


o 


o 


o 


o 


8. Chemist 


o 


o 


o 


o 


9. Engineer 


o 


o 


o 


o 


10. Accountant 


o 


o 


o 


o 


11. Biologist 


o 


o 


o 


o 


12. Architect 


o 


o 


o 


o 
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5. About Yourself 

How well do you expect to do this year in your: 





Not very 

well 

1 


OK/Pretty 

well 

2 


Very well 
3 


English class? 


O 


O 


O 


Math class? 


O 


o 


o 


Science class? 


o 


o 


o 



Year of Birth: 

BoyO Girl O 

My family is: (CHECK ONE ONLY) 

O African American 
O Caucasian/ White 

O Asian (Please Specify): 

O Hispanic (Please Specify): 

O Biracial/ Multiracial (Please Specify): 
O Other (Please Specify): 



Do you plan to go to college? Yes O No O Not sure O 






47 



Post-test 



The Post-test Questionnaire was identical to the pre-test except for the addition of the 
following question, the answers to which required having participated in the intervention. 



Please tell us how interesting you found the activities on engineering in this class: 
Not at all interesting O Somewhat interesting O Very interesting O 
Please explain your answer. 



Thanks for Your Help! 
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Appendix C 

Psychometric Analysis of the Attitude Scales 

Reliability ( Cronbach ’s alpha ) 

Internal consistency scores of scales indicate the degree to which all items in the scale are 
measuring the same phenomenon, in this case, self-esteem. The internal consistency score is 
often reported as Chronbach’s alpha, which varies from .00 to .99. In general, the more items 
that measure the same phenomenon, the closer the scales internal consistency will be to .99. 
As can be seen in Table 1, the scales have adequate internal consistency; they are all above 
.80. 

Table 1: Unstandardized Cronbach’ s alpha internal consistency scores for each scale at both 
pre- and post-test for the data set used in the current analyses. 



Scale 


Number 
of Items 


Number 

of 

Subjects 


Pre-Test 

Cronbach’s 

Alpha 


Number 

Of 

Subjects 


Post- Test 
Cronbach’s 
Alpha 


Engineering 


12 


396 


.829 


409 


.842 


Math 


13 


394 


.866 


392 


.880 


Science 


10 


400 


.865 


406 


.858 



Validity 

The set of correlations in Table 5 below provides information related to the construct validity 
of the attitude surveys. If the items in each of the surveys measure the underlying constructs 
adequately, we would expect that the three surveys would be moderately correlated with each 
other (r =.25 - .50) range but that the pre -post-intervention administrations of each subject 
would be strongly correlated with each other (r=.60 and above range). As predicted, the 
results show that the highlighted correlation coefficients between the pre- and post- 
intervention scores are all above r=.50. The correlations among the surveys are weaker. The 
fact that the prediction was supported suggests that the measures are very likely measuring 
the underlying constructs they were designed to measure; hence they have construct validity. 

The only exceptions are the high correlations between pre-intervention engineering and pre- 
intervention science scores (r=.552). This can be viewed as an indicator of the closer 
association of engineering and science at pre-tests, as the engineering curriculum was being 
implemented in science classes. 
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Table 2: Correlations ( Pearson r ) among average pre- and post-intervention attitude 
survey scores. 





Pre 

Eng. 


Post 

Eng. 


Pre Math 


Post 

Math 


Pre 

Science 


Pre-intervention engineering 
attitude scores 


1 










Post-intervention engineering 
attitude scores 


.699 


1 








Pre-intervention math 
attitude scores 


.411 


.363 


1 






Post-intervention math 
attitude scores 


.347 


.452 


.703 


1 




Pre-intervention science 
attitude scores 


.552 


.434 


.339 


.249 


1 


Post-intervention science 
attitude scores 


.490 


.490 


.320 


.328 


.637 



